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INSPECTOR  GENERAL 

DEPARTMENT  OF  DEFENSE 
400  ARMY  NAVY  DRIVE 
ARLINGTON,  VIRGINIA  22202-2884 


February  22,  1995 

MEMORANDUM  FOR  DIRECTOR,  OPERATIONAL  TEST  AND  EVALUATION 

ASSISTANT  SECRETARY  OF  THE  NAVY  FINANCIAL 
MANAGEMENT) 

SUBJECT;  Congressionally  Requested  Audit  of  the  Airborne  Self-Protection  Jammer 
(Report  No.  95-128) 


We  are  providing  this  final  report  for  your  review  and  comments.  Senator 
David  Pryor  requested  this  audit  of  the  Airborne  Self-Protection  Jammer  (ASPJ)  in 
April  1994.  The  audit  addressed  the  Navy  plans  to  test  and  install  the  ASPJ  on  the 
F-14D  aircraft,  the  appropriateness  of  DoD  support  for  the  integration  of  the  ASPJ 
through  Foreign  Militmy  Sales,  and  DoD  efforts  to  identify  cost-effective  self- 
protection  systems  for  common  use  on  Navy  and  Air  Force  aircraft.  Comments  on  a 
draft  of  this  report  were  considered  in  preparing  the  final  report. 

DoD  Directive  7650.3  requires  that  all  audit  recommendations  be  resolved 
prompUy.  Therefore,  we  request  that  the  Director,  (^rational  Test  and  Evaluation; 
the  Commander,  Operational  Test  and  Evaluation  Force;  and  the  F-14D  Program 
Manager  provide  comments  on  the  unresolved  recommendations  by  April  24,  1995. 

We  appreciate  the  courtesies  extended  to  the  audit  staff.  If  you  have  questions 
on  this  audit ,  please  contact  Mr.  John  E.  Meling,  Program  Director,  at  (703)  604- 
9091  (DSN  664-9091)  or  Mr.  Brian  M.  Flynn,  Project  Manager,  at  (703)  604-9076 
(DSN  664-9076).  Appendix  F  lists  the  distribution  of  this  report.  The  audit  team 
members  are  listed  inside  the  back  cover. 


Assistant  Inspector  General 
for  Auditing 


Office  of  the  Inspector  General,  DoD 


Report  No.  95-128 
(Project  No.  4AE-5036) 


February  22, 1995 


THE  CONGRESSIONALLY  REQUESTED  AUDIT  OF  THE 
AIRBORNE  SELF-PROTECTION  JAMMER 


EXECUTIVE  SUMMARY 


Introduction.  The  Airborne  Self-Protection  Jammer  (ASPJ)  is  an  electronic 
countermeasure  system.  Operational  test  and  evaluation  of  the  ASPJ  system,  using  the 
Navy's  F/A-18  aircraft,  determined  that  the  ASPJ  was  not  operationally  suitable  or 
operationally  effective,  resulting  in  termination  of  production  contracts  in  December 
1992.  However,  the  Navy  plans  to  use  the  ASPJ  on  53  F-14D  aircraft  if  operational 
tests  show  that  the  aircraft  are  more  survivable  with  ASPJ  than  without  it.  The  F-14D 
aircraft  operational  tests  are  scheduled  for  FY  1995.  The  Under  Secretary  of  Defense 
for  Acquisition  and  Twhnology  supports  ASPJ  deployment  if  it  contributes  to  F-14D 
survivability.  In  addition,  the  DoD  plans  to  assist  the  foreign  sales  of  the  ASPJ 
through  ASPJ  software  integration  on  F/A-18  and  F-16  aircraft. 

Objectives.  We  initiated  this  audit  at  the  request  of  Senator  David  Pryor.  He  asked  us 
to  evaluate  Navy  plans  to  test  and  install  the  ASPJ  on  the  F-14D  aircraft,  review  the 
appropriateness  of  DoD  integration  of  ASPJ  software  through  Foreign  Military  Sales, 
and  review  DoD  efforts  to  identify  cost-effective  self-protection  systems  for  common 
use  on  Navy  and  Air  Force  aircraft. 

Au^t  Results.  DoD's  plans  to  integrate  commercially  purchased  ASPJs  into  foreign 
military  sales  purchased  F/A-18  and  F-16  aircraft  were  deemed  appropriate.  Further, 
DoD  plans  to  identify  cost-effective  electronic  warfare  self-protection  systems  for 
common  use  on  next-generation  Navy  and  Air  Force  aircraft  are  in  progress. 
However,  the  Navy's  operational  test  plan  for  the  F-14D  aircraft  does  not  contain 
measures  of  effectiveness  to  make  an  objective  assessment  as  to  whether  the  ASPJ 
makes  a  positive  contribution  to  F-14D  aircraft  survivability.  As  a  result,  the  ASPJ 
could  be  deployed  to  the  F-14D  aircraft  fleet  without  the  Navy  knowing  whether  the 
ASPJ  does  make  a  positive  contribution  to  F-14  aircraft  survivability.  Because  the 
plann^  F-14D  aircraft^  operational  test  is  not  designed  to  determine  the  operational 
effectiveness  and  suitability  of  the  ASPJ,  DoD  decisionmakers  will  not  be  able  to  use 
the  test  results  to  support  future  decisions  (not  now  planned)  to  produce  additional 
ASPJs  or  install  the  ASPJ  on  aircraft  other  than  the  F-14D. 

Internal  Controls.  Internal  controls  were  not  assessed  because  time  constraints 
dictated  that  we  focus  solely  on  the  questions  posed  by  the  requestor. 

Potential  Benefits  of  Review.  Potential  benefits  for  this  audit  are  nonmonetary. 
Potential  benefits  of  the  audit  are  in  Appendix  D. 

Summary  of  Recommendations.  We  recommended  that: 

o  the  F-14D  Program  Manager,  in  conjunction  with  the  Commander, 
Operational  Test  and  Evaluation  Force,  revise  the  F-14D  aircraft  Test  and  Evaluation 
Master  Plan  and  F-14D  Aircraft-Dedicated  Operational  Test  Plan  to  include  specific 
ASPJ  measures  of  effectiveness  and 
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p  i.ir»  ^rational  Test  and  Evaluation,  withhold  approval  of  the 

Tff^tivelS^eSd^  Evaluation  Master  Plan  until  specific  ASPJ  measures  of 

Management  Comments.  The  Commander,  Operational  and  Test  Evaluation  Force 
agree  to  revise  the  F-14D  test  plans  to  establish  discrete  ASPJ  measures  of 
effectiveness  for  determnung  whether  the  ASPJ  makes  a  positive  contribution  to  F-14D 

wustactoiy.  Ins^d,  the  Commander  said  that  testers  would  collect  discrete 

•?h  'Whether  the  F-14D  aircraft  is  more  survivabte  witti 

the  jammer  than  without  it  and  discrete  suitability  testing  to  characterize  ASPJ's  built-in 
test  performaitce.  The  Director,  Operationaf  Test  Ind  EvaluattoT  Mted  £  thS 
rewmmended  measures  of  effectiveness  were  not  applicable  to  the  scope  and  olanned 
Sve  S,?#  oparatton^test  phase.  ‘?heref£’£sff£  hfTrdI 

F-14D  test  plans  if  the  Navy  provided  for  collection  of  discrete  ASPJ 
effectiveness  and  suitability  data  as  stated  above.  Part  U  contains  a  discussion  of 
manageme^  s  comments  to  the  report.  Part  IV  contains  the  complete  texts  of 

wmments  from  the  Naval  Commander,  Operational  Test  and  Evaluation  Force  and  the 
Director,  Operational  Test  and  Evaluation.  i^vaiuaaon  t-orce,  and  the 

recommendation  in  response  to  management 
recommendations  as  written.  Unless  the  Navv 
“d  meaningful  ASPJ  measures  of  effectiveness  in  F-%  tes^t 

K’lJSeTtiie  not  be  able  to  make  an  objective  assessment 

f  i  wu  ^  ^  positive  contiibution  to  F-14D  aircraft  survivabilitv 

demonstrated  ASPJ  built-in-test  performance  is  satisfactory  at  the  end 
Oj^ratioi^  test.  We  request  that  Ae  Director,  Operational  Test  and 
F  UD  Pm’aJLm  Commander,  Operational  Test  and  EvaluXn  Force;  and  the 
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Part  I  -  Introduction 


Background 


Self-Protection  Jammer  (ASPJ)  is  an  electronic  countermeasure 
system  ongmally  designed  for  use  on  multiple  aircraft  types.  The  ASPJ  is 
increase  mrcraft  si^ivability  by  jamming  or  deceiving  radar- 
controlled  weapons  of  the  enemy  by  projecting  a  false  location  of  the  aircraft. 

pie  operation^  test  and  evaluation  of  the  ASPJ  was  conducted  from 

August  1991  trough  Apnl  1992,  using  the  Navy’s  F/A-18  aircraft.  The 
Coi^ander,  Operational  Test  and  Evaluation  Force  (COMOPTEVFORl  and 
the  Director,  Operational  Test  and  Evaluation  (OT&E),  Office  of  the  Sewetary 
Acm  t®st  results.  The  organizations  concluded  that  the 

opeJ^donally  suitable  Md  the  Director,  OT&E,  also  concluded 
thanhe  ASPJ  was  not  operationally  effective.  As  a  result,  the  Under  Secretary 
Acquisition  and  Technology  (USD[A&T1)  directed  the  Navy  to 
cancel  ASPJ  production  contracts.  The  International  Telephone  and  Telegraph 
Westinghouse  Electric  Corporation  developed  and  product 

pt<hcS95»„1ffof.S'’"  “ 

contracts  affected  the  electronic  warfare 
(EW)  suite  planed  for  the  F-14D  aircraft,  which  was  designed  to  use  the  ASPJ 

decided  to  test  and  use  the  95  already 
procui^  ASPJ  urats  for  its  53  F-14D  aircraft.  The  production  contracts' 
^^llation  also  effected  Foreign  MiUtary  Sales  (FMS).  Korea,  Finland,  Sd 
Smtrerland  to  procure  ASPJs  for  F/A-18  and  F-16  aircraft  they 

a^uired  through  FMS  contracts.  Instead  of  procuring  ASPJ  hardware  through 
FMS,  these  countnes  now  plan  to  procure  the  hardware  directly  from  the 
m^ufacmrers.  For  security  reasons,  the  U.S.  Government  will  provide  the 
ASPJ  software  integration  through  FMS.  ^ 

of  the  Department  of  Defense*  released  $7.8  miUion 
A  om  •  Development,  Test  and  Evaluation  funding  for  testing 

ASPJ  aircraft.  With  the  funding,  the  Navy  procured  six  modified 

ASPJ  fom^d  and  aft  installation  racks  and  interim  support  financing.  The 

oArati^Sin“n'®»5.““" 

^e  Npy  does  not  plM  to  a^uire  additional  ASPJs  for  future  upgrades  to  the 
F-14D^Srcr^r^"  operationally  effective  and  Siitable  on  the 


*Renamed  Under  Secretary  of  Defense  (Comptroller)  in  October  1994. 


Litroduction 


Objectives 


We  imtiat^  this  audit  at  the  request  of  Senator  David  Pryor  He  asked  us  to 
evaluate  the  Navy  plans  to  test  and  install  the  ASPJ  oh  the  F?14D  drcAft 

of  poJ?^integration  of  ASPJ  software  through  Fl^’ 
and  review  DoD  efforts  to  identify  cost-effective  self-protection  systems  for 
common  use  on  Navy  and  Air  Force  aircraft.  ApJeS  A  Z 

complete  text  of  Senator  Pryor's  letter.  Pari  H  a  ZS  ^d 

recommendations  on  the  Operational  Testing  of  the  Airborne  Llf-ffictiSn 
.Api^ndixB  addresses  DoD  Efforts  to  Identify  Cost-Effective 
Se^rotection  Systems  for  Common  Use.  The  section,  "Prior  AudKd 

of  ASptSSe  ””  of  tategralion 


Scope  and  Methodology 


We  performed  this  prog^  results  audit  from  April  through  October  1994  and 
review^  rwords  dated  from  July  1991  through  October  1994.  We  oerformed 
ttis  audit  m  a^rd^^  with  auditing  standards  issued  by  the  Comptroller 
G^eral  of  the  United  States,  as  implemented  by  the  Inspector  General^  DoD 


toe  operational  testing  and  deployment  of 

me  Abrj  on  F-14D  aircraft  and  appropriateness  of  TJ  SI 

“Oiaft.  We  reviLS  Md  <SS  the 
F  14D  dedicated  oj^ratiqnal  test  and  evaluation  test  plan  and  toe  F-14D  test  and 

determine-  the  F-14D  program  office  and  COMOPTEVFOR  to 

o  toe  extent  of  testing  dedicated  to  ASPJ; 

show  wUS?r  “ 

tiiscuss^  the  appropriateness  of  DoD  support  for  the  integration  of  toe 
ASPJ  through  toe  Foreign  Military  Sales  program  with  toe  Navy^  Tactical  Air 
Warfare,  F-14D,  and  F/A-18  program  offices  and  int^ewed  DoD 
and  Navy  personnel  responsible  for  implementing  FMS  policy.  Because  of  time 
wiwtrmnts,  we  conducted  a  status  review  of  relevant  programs  associated  with 

M  efforts  to  idenUfycost-effecttve  self-p«>teetion™  Kfar  SS^s^S 

Navy  Md  Air  Force  aircraft.  We  did  not  rely  on  cimputerSiSS 
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Introduction 


Internal  Controls 


Internal  controls  were  not  assessed  because  time  constraints  dictated  that  we 
focus  solely  on  the  questions  posed  by  the  requestor. 


Prior  Audits  and  Other  Reviews 


The  General  Accounting  Office  (GAO)  issued  a  report  in  June  1994  that 
address^  our  audit  objective  concerning  the  appropriateness  of  DoD  integration 
software  through  FMS.  The  results  of  our  audit  support  the 
conclusions  drawn  by  the  GAO  m  that  report.  One  other  report  by  fte  GAO 

^d  one  by  the  Inspector  General,  DoD,  relating  to  ASPJ  development  are 
discussed  in  Appendix  C.  y  i 
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Part  II  -  Finding  and  Recommendations 


Operational  Testing  of  the  Airborne 
Self-Protection  Jammer 

The  Navy's  operational  test  plan  for  the  F-14D  aircraft  does  not  contain 
measures  of  effectiveness  to  determine  whether  the  ASPJ  contributes  to 
F-14D  survivability.  The  F-14D  operational  test  plan  was  designed  to 
measure  the  collective  contributions  all  F-14D  electronic  warfare 
systems  made  to  determine  aircraft  survivability,  mission  effectiveness, 
and  suitability.  As  a  result,  the  ASPJ  could  be  deployed  on  F-14D 
aircraft  without  the  Navy  Imowing  whether  the  ASPJ  does  make  a 
positive  contribution  to  F-14D  aircr^  survivability. 


Background 


Congressional  Concern  About  ASPJ  Testing  as  Part  of  the  F-14D  Aircraft 
Operational  Test.  On  August  5,  1993,  Senator  Pryor  raised  concerns  to  the 
Secretary  of  Defense  about  operational  testing  and  deployment  of  the  ASPJ  on 
F-14D  aircraft  in  view  of  the  ASPJ  system  failing  the  operational  tests  and  the 
termination  of  the  program  in  1992.  On  September  8,  1993,  the  USD(A&T) 
responded  to  the  Senator  and  stated  that  he  supported  operational  testing  and 
subsequent  deployment  of  the  ASPJ  system  on  the  F-14D  aircraft  if  that  aircraft 
was  assessed  more  survivable  with  the  EW  suite  than  without  it  and  the  ASPJ 
was  determined  to  be  supportable  in  the  fleet.  On  March  22,  1994,  the  Deputy 
Under  Secretary  for  Defense  (Acquisition  Reform)  amplified  this  policy  during 
congressional  hearings  stating; 

As  part  of  the  F-14D  operational  evaluation,  it  [the  F-14D  aircraft] 
will  be  tested  with  the  ASPJ.  If  it  is  determined  that  die  F-14D  is 
more  survivable  with  the  ASPJ  than  without  it,  it  will  be  installed  for 
fleet  use.  The  Deputy  Secretary  of  Defense,  however,  has  committed 
to  instruct  the  Navy  to  withhold  any  F-14D  testing  of  this  system 
until  a  Director  of  Operational  Test  and  Evaluation  has  been 
confirmed,  is  on  the  job,  and  offers  recommendations  as  to  the 
adequacy  and  utility  of  such  tests. 

On  October  3,  1994,  the  Senate  confirmed  the  President's  nominee  for 
Director,  OT&E. 

DoD  Testing  Policy.  DoD  Instruction  5000.2,  "Defense  Acquisition 
Management  Policies  and  Procedures,"  February  23,  1991,  requires  that  the 
Director,  OT&E,  approve  the  ad^uacy  of  OT&E  plans  for  all  major  Defense 
acquisition  programs  before  the  initiation  of  operational  testing.  The  Instruction 
also  requires  that  all  material  deficiencies  identified  during  operational  testing 
be  corrected  before  system  deployment  and  the  Military  Department's 
independent  test  organization  verify  those  corrections  during  follow-on  OT&E. 
Further,  the  Instruction  requires  that  the  system  developer  certify  that  the 
system  is  ready  for  the  dedicated  phase  of  OT&E  before  the  Military 
Dq)artment's  independent  test  organization  conducts  the  operational  test. 
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Operational  Testing  of  the  Airbone  Self-Protection  Januner 


Navy  Testing  Policy.  Navy  Instruction  5440. 47F,  "Mission  and  Functions  of 
Operational  Test  and  Evaluation  Force,"  May  21,  1984,  assigns 

COMOPTEVFOR,  the  Navy's  indq)endent  test  organization,  with  responsibility 
for: 


o  planning  and  conducting  OT&E  in  a  realistic,  threat-representative 
operational  environment; 

o  conducting  follow-on  OT&E  to  verify  resolution  of  problems  revealed 
during  OT&E  and  to  demonstrate  that  operational  suitability  criteria,  such  as 
reliability,  maintainability,  and  availability,  are  actually  achieved;  and 

o  developing  tactics  and  procedures  for  employment  of  systems  that 
undergo  OT&E. 


ASPJ  Improvements  and  F-14D  Operational  Test  Plans 


Navy  Efforts  to  Improve  ASPJ's  Operational  Performance  Since  1992.  In 
1992,  COMOPTEVFOR  concluded  that  the  ASPJ  was  not  operationally  suitable 
during  operational  tests  because  the  ASPJ  failed  critical  operational  issues  such 
as  reliability,  human  factors,  and  built-in-test  (BIT),  all  due  to  BIT  deficiencies. 
COMOPTEVFOR  also  concluded  that  the  Navy  needed  to  improve  ASPJ's 
effectiveness  and  correct  the  BIT  deficiencies  before  ASPJ  deployment.  On 
December  4,  1992,  the  Director,  OT&E,  evaluated  the  same  ASPJ  operational 
test  results  and  concluded  that  in  addition  to  COMOPTEVFOR' s 
determinations,  the  ASPJ  was  not  operationally  effective  when  compared  to 
aircraft  with  and  without  existing  self-protection  jammers. 

When  incorporating  the  ASPJ  in  the  F-14D  aircraft,  the  Navy  changed  the 
F-14D  airci^t  and  ASPJ  configurations  that  may  resolve  some  of  the  ASPJ 
operational  effectiveness  and  suitability  performance  deficiencies  identified 
during  ASPJ's  1992  operational  tests.  Configuration  changes  include: 

o  installing  ASPJ  forward  and  aft  installation  racks  designed  to 
specifically  fit  the  F-14D  aircraft  (the  Navy  believes  that  the  racks  will 
eliminate  many  BIT  failures  noted  in  tiie  ASPJ  operational  test  that  were  caused 
by  racks  fitting  poorly  in  the  F/A-18  test  aircraft); 

o  updating  the  ASPJ  user  data  file  concerning  the  threat  library  and 
electronic  countermeasure  techniques  and  incorporating  the  ASPJ  with  improved 
BIT  software  in  the  F-14D  operational  flight  program; 

o  incorporating  multifilter  assemblies  in  each  quadrant  of  the  F-14D 
aircraft  to  permit  the  aircraft  radar  and  the  ASPJ  to  share  frequency  bands  to 
carry  out  their  assigned  missions  without  interference; 
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Operational  Testing  of  the  Airborne  Self-Protection  Januner 


o  adding  an  aft  high  band  transmitter  and  augmentation  receiver  to  the 
ASPJ;  and 

o  modifying  the  F-14D  airframe  and  antenna. 

The  impact  of  these  configuration  changes  on  ASPJ's  operational  effectiveness 
and  suitability  will  not  be  known  until  the  F-14D  aircraft-dedicated  OT&E  is 
conducted.  Through  October  1994,  the  F-14D  aircraft  developer  had  not 
certified  that  tire  configuration  changes  were  ready  for  the  F-14D  aircraft 
dedicated  OT&E. 

Draft  F-14D  Aircraft  Test  and  Evaluation  Master  Plan.  In  March  1994,  the 
Navy  Program  Executive  Officer  for  Tactical  Air  and  COMOPTEVFOR  revised 
the  F-14D  aircraft  test  and  evaluation  master  plan  (TEMP)  in  response  to 
direction  from  the  USD(A&T)  and  the  Chief  of  Naval  Operations.  TTie  draft 
TEMP  would  require  COMOPTEVFOR  to  assess  the  ASPJ  for  its  contribution 
to  mission  success  as  an  element  of  the  F-14D  aircraft  EW  suite  during  the 
F-14D  aircraft-dedicated  OT&E.  The  EW  suite  consists  of  the  ASPJ,  the 
ALE-39  countermeasure  dispenser  system,  and  the  ALR-67  radar  warning 
receiver.  The  draft  F-14D  aircraft  TEMP  is  pending  approvals  from  the 
Director,  OT&E,  and  the  Director,  Test  and  Evaluation,  Office  of  the 
USD(A&T). 

F-14D  Aircraft-Dedicated  Operational  Test  and  Evaluation  Test  Plan.  As  a 
result  of  revising  the  F-14D  aircraft  TEMP,  COMOPTEVFOR  revised  the 
F-14D  aircraft-dedicated  OT&E  Test  Plan  (F-14D  Test  Plan)  to  add  the  test 
objective  for  determining  whether  the  aircraft  EW  suite  contributes  to  mission 
accomplishment  in  a  threat  system  environment  and  increases  aircraft 
survivability.  In  addition,  the  F-14D  Test  Plan  indicates  that  testers  wiU 
determine  the  effectiveness  of  aircraft  maneuvers  alone  as  a  defense  against 
threat  missiles. 

The  F-14D  Test  Plan  states  that  a  minimum  of  18  sorties,  approximately 
32.4  flight  hours,  will  be  allocated  to  collect  EW  effectiveness  data  necessary  to 
measure  the  ASPJ's  contribution  to  F-14D  aircraft  survivability.  Each  F-14D 
aircraft  sortie  will  last  from  1.3  to  1.8  hours  and  will  use  one  of  six  different 
combinations  of  EW  defensive  techniques.  During  each  sortie,  the  testers  plan 
to  collect  data  on  the  effectiveness  of  each  subsystem  of  the  EW  suite  and 
aircraft  maneuvers  by  measuring  their  contribution  to  defeating  ground  threat 
radars.  The  F-14D  Test  Plan  indicates  that  another  60  sorties  will  be  flown 
during  power  projection  missions  and  threat  detection  and  tracking  missions  that 
periodically  will  also  allow  the  testers  to  measure  the  effectiveness  of  the  EW 
techniques  as  a  secondary  test  objective. 

The  Commander,  Air  Test  and  Evaluation  Squadron  Four,  is  responsible  for  the 
execution  of  the  dedicated  F-14D  aircraft-dedicated  operational  test  and 
evaluation.  In  terms  of  BIT  testing,  the  Squadron's  F-14D  operational  test 
director  stated  that  the  ASPJ  would  be  tested  before,  during,  and  after  each 
sortie  except  when  the  operational  test  director  decides  that  the  ASPJ  will 
interfere  with  certain  test  objectives  and  when  the  ASPJ  is  undergoing 
maintenance. 
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The  table  summarizes  the  number  of  sorties  dedicated  to  the  collection  of  ASPJ 
effectiveness  and  suitability  data  as  conducted  during  the  1992  ASPJ-dedicated 
OT&E  and  the  planned  F-14D  aircraft-dedicated  OT&E. 


Sorties  Allocated  for  ASPJ  Data  Collection 


(1.3  to 

1.8  hours  per  sortie) 

During  1992 

Planned  During 

ASPJ- 

F-14D  Aircraft- 

Sorties 

Dedicated  OT&E 

Dedicated  OT&E 

ASPJ  Effectiveness  Sorties 

63 

18 

ASPJ  Suitability  Sorties 

200+ 

200+ 

Also,  the  F-14D  Test  Plan  states  that  testers  will  consult  the  ASPJ  test  report  for 
the  1992  OT«&E  to  assist  in  answering  whether  or  not  similar  ASPJ  results  were 
observed  during  the  F-14D  aircraft-d^icated  OT&E. 

Offlce  of  the  Director,  Operational  Test  and  Evaluation,  Assessment  of  the 
F-14D  Test  Plan.  The  Military  Assistant  for  Air  Warfare  Systems  believed  that 
the  planned  EW  tests  in  the  F-14D  Test  Plan  will  collect  sufficient  effectiveness 
data  necessary  to  measure  the  ASPJ's  contribution  to  F-14D  aircraft 
survivability.  However,  he  stated  that  the  18  dedicated  EW  sorties  are  probably 
not  sufficient  to  make  a  statistical  comparison  with  the  ASPJ  operational  test 
results  from  1992.  He  added  that  for  comparison  purposes,  the  planned  tests 
are  sufficient  to  confirm  positive  or  negative  trends  in  ASPJ  performance.  He 
also  believed  that  200  or  more  flight  hours  of  ASPJ  suitability  testing  could 
determine  whether  improvements  in  BIT  performance  were  made. 


Determining  Whether  ASPJ  Effectiveness  Contributes  to 
F-14D  Survivability 

As  drafted,  the  F-14D  Test  Plan  will  enable  COMOPTEVFOR  to  collect  data  to 
measure  whether  the  ASPJ  contributes  to  F-14D  aircraft  survivability. 
However,  the  F-14D  Test  Plan  did  not  contain  a  measure  of  effectiveness  for 
use  by  testers  in  determining  whether  the  ASPJ  makes  a  positive  contribution  to 
F-14D  survivability.  This  condition  was  caused  by  COMOPTEVFOR 
designing  the  F-14D  Test  Plan  to  measure  the  collective  contributions  all  F-14D 
aircraft  EW  systems  made  instead  of  measuring  individual  EW  system 
contributions  to  determine  aircraft  survivability. 

Measure  of  Effectiveness.  The  F-14D  Test  Plan  did  not  include  a  specific 
ASPJ  measure  of  effectiveness  similar  to  one  established  for  the  1992  ASPJ 
dedicated  OT&E  for  determining  whether  the  ASPJ  contributes  to  F-14D 
aircraft  survivability. 
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For  the  1992  OT&E,  COMOPTEVFOR  assessed  ASPJ  performance  for: 

o  responding  in  a  timely  manner  and  confusing  threat  radar  operators, 
by  measuring  the  probability  of  threat  shots  taken,  and 

o  degrading  and  denying  threat  tracking,  by  measuring  the  probability 
of  threat  shots  taken  that  resulted  in  hits. 

COMOPTEVFOR  used  the  determined  probabilities  to  compute  an  ASPJ 
measure  of  effectiveness  for  aircraft  survivability.  However,  the  F-14D  test 
plan  requires  ASPJ  data  be  collected  as  part  of  the  EW  suite  without  separately 
computing  an  aircraft  survivability  factor  based  on  ASPJ  performance.  Based 
on  the  Deputy  Under  Secretary  for  Defense  (Acquisition  Reform) 's  commitment 
to  Congress,  COMOPTEVFOR  needs  to  establish  a  sp^ific  ASPJ  measure  of 
effectiveness  to  determine  whether  the  F-14D  aircraft  is  more  survivable  with 
the  ASPJ  than  without  it  before  the  ASPJ  is  deployed  for  fleet  use. 

Comparison  of  ASPJ  Test  Results.  The  F-14D  Test  Plan  states  that  the  testers 
will  consult  the  ASPJ  test  report  for  the  1992  OT&E  to  assist  in  answering 
whether  or  not  similar  ASPJ  results  were  observed  during  the  F-14D 
aircraft-dedicated  OT&E.  In  our  opinion,  this  comparison  is  important  because 
the  testers  can  determine  the  extent  that  ASPJ  contributes  to  F-14D  aircraft 
survivability.  Additionally,  testers  must  clearly  state  the  results  of  the 
comparison  in  the  ensuing  test  report  to  dispel  potential  criticism  that  the  ASPJ 
was  not  adequately  tested  before  being  deployed  to  the  F-14D  aircraft  fleet. 


Measuring  ASPJ  Built-In-Test  Performance 


As  drafted,  the  F-14D  Test  Plan  will  enable  COMOPTEVFOR  to  collect 
sufficient  ASPJ  data  on  BIT  performance.  Poor  BIT  performance  was  the 
primary  deficiency  identified  during  the  1992  OT&E.  However,  the  F-14D 
Test  Plan  did  not  contain  a  specific  ASPJ  measure  of  effectiveness  for 
determining  an  acceptable  level  of  performance  for  ASPJ  BIT  suitability.  This 
condition  was  caused  by  COMOPTEVFOR  designing  the  F-14D  Test  Plan  to 
measure  the  collective  contributions  all  F-14D  EW  systems  made  to  determine 
F-14D  aircraft  survivability,  mission  effectiveness,  and  suitability. 
COMOPTEVFOR  stated  that  as  a  subsystem  of  the  aircraft,  ASPJ  is  included  in 
the  list  of  avionics  work  unit  codes  and,  therefore,  will  be  evaluated  for 
operational  suitability. 

COMOPTEVFOR  stated  that  it  was  not  possible  to  compare  ASPJ's  BIT 
performance  on  the  F/A-18  aircraft  (used  for  ASPJ  testing  in  1992)  to  the  BIT 
performance  on  the  F-14D  aircraft  because  of  differences  in  aircraft  size  and 
configuration.  Although  a  direct  comparison  of  BIT  failure  rates  may  not  be 
relevant  in  determining  the  degree  of  improvements  to  the  ASPJ  BIT 
performance,  an  ASPJ  measure  of  effectiveness,  including  BIT  performance,  is 
necessaty  so  testers  can  objectively  comment  on  ASPJ  BIT  performance  at  the 
completion  of  the  F-14D  aircraft-dedicated  OT&E. 
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Conclusion 


Test  data  to  be  collected  during  the  F-14D  aircraft-dedicated  OT&E  should 
provide  COMOPTEVFOR  sufficient  test  data  to  measure  ASPJ’s  contribution  to 
F-14D  aircraft  survivability,  mission  effectiveness,  and  suitability.  However, 
objective  and  meaningful  measures  of  effectiveness,  for  determining  whether  the 
ASPJ  makes  a  positive  contribution  to  F-14D  aircraft  survivability  and  for  ASPJ 
BIT  performance,  need  to  be  in  the  F-14D  Test  Plan  before  the  F-14D 
aircraft-dedicated  OT&E  is  carried  out.  Because  the  planned  F-14D  aircraft 
operational  test  is  not  designed  to  determine  the  operational  effectiveness  and 
suitability  of  the  ASPJ,  DoD  decisionmakers  will  not  be  able  to  use  the  test 
results  to  support  future  decisions  (not  now  planned)  to  produce  additional 
ASPJs  or  ins^l  the  ASPJ  on  aircraft  other  than  the  F-14D. 


Recommendations,  Management  Comments,  and  Audit 
Responses 


1.  We  recommend  that  the  F-14D  Program  Manager,  in  copjunction  with 
the  Commander,  Operational  Test  and  Evaluation  Force,  revise  the  draft 
F-14D  aircraft  Test  and  Evaluation  Master  Plan  and  F-14D 
Aircraft-Dedicated  Operational  Test  and  Evaluation  Test  Plan  to  include 
specific  Airborne  Self-Protection  Jammer  measures  of  effectiveness  for 
determining: 

a.  Whether  the  Airborne  Self-Protection  Januner  makes  a  positive 
contribution  to  F-14D  aircraft  survivability. 

b.  Airborne  Self-Protection  Jammer  mission  suitability,  including 
built-in-test  performance. 

Commander,  Operational  Test  and  Evaluation,  Comments.  The 
Commander  did  not  concur  with  the  recommendation.  He  stated  that  the  F-14D 
Test  Plan  contains  three  measures  of  effectiveness  that  will  allow 
COMOPTEVFOR  to  specifically  address  ASPJ  effectiveness.  He  stated  that  the 
measures  of  effectiveness  (shot  denial,  missile  lethality,  and  shot  degradation) 
are  similar  to  that  collected  in  the  earlier  dedicated  ASPJ  operation^  test  and 
evaluation.  He  also  stated  that  although  no  specific  subsystems  are  mentioned 
in  the  F-14D  Test  Plan,  BIT  performance  will  be  measured  for  all  applicable 
avionic  subsystems.  Accordingly,  he  believes  that  sufficient  performance  data 
will  be  collected  to  support  specific  F-14D/ASPJ  comments  and 
recommendations  in  the  report  issu^  on  the  F-14D  operational  test  and 
evaluation. 
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Based  on  recommendations  from  the  F-14D  Test  Planning  Working  Group,  the 
Commander  stated  that  the  F-14D  TEMP  and  Test  Plan  were  being  updat^  to 
include  the  following  critical  operational  issue:  "Is  the  F-14D  more  survivable 
with  ASPJ  as  a  part  of  the  EW  suite  than  without  it?" 

The  Commander  also  stated  that  the  recommendations  should  be  addressed  to 
the  F-14D  Program  Office  because  only  the  F-14D  Program  Office  has 
authority  for  updating  the  TEMP. 

Audit  Response.  Management  comments  were  not  fully  responsive  to  the 
intent  of  Recommendations  l.a.  and  l.b.  Establishing  a  critical  operational 
issue  in  the  TEMP  that  states  "Is  the  F-14D  more  survivable  with  ASPJ  as  a 
part  of  the  EW  suite  than  without  it?"  is  a  step  in  the  right  direction.  However, 
COMOPTEVFOR  has  not  taken  the  next  step  to  enable  an  objective  answer  to 
this  critical  operational  issue,  that  is,  establishing  separate  (discrete)  ASPJ  levels 
of  performance  or  thresholds  for  Ae  measures  of  effectiveness  to  determine 
whether  the  ASPJ  makes  a  positive  contribution  to  F-14D  survivability  and 
whether  the  ASPJ's  BIT  performance  is  satisfactory  for  use  on  the  F-14D 
aircraft.  Under  the  current  test  plan  methodology,  EW  effectiveness  tests  will 
measure  the  combined  effects  of  radar  warning,  expendables,  self-protection 
jamming,  and  aircraft  maneuvering.  Further,  though  EW  suitability  tests  will 
record  ASPJ  BIT  performance  separately,  the  results  will  be  assessed  as  part  of 
the  overall  F-14D  BIT  performance. 

However,  establishing  measures  of  effectiveness  is  important  because 
decisionmakers  need  to  have  an  objective  benchmark  as  to  what  degree  of  added 
F-14D  aircraft  survivability  provided  by  the  ASPJ  warrants  the  costs  of 
installing,  maintaining,  and  operating  the  system.  Second,  it  is  just  as  important 
for  decisionmakers  to  know  what  is  the  minimum  ASPJ  BIT  performance 
needed  that  warrants  the  costs  of  installing,  maintaining,  and  operating  the 
ASPJ.  To  do  this,  the  F-14D  Program  Manager  needs  to  analyze  ASPJ  life- 
cycle  and  support  costs  relative  to  the  increase  in  F-14D  survivability  as 
demonstrated  in  the  F-14D  operational  test. 

Based  on  management  comments,  we  readdressed  Recommendations  l.a.  and 
l.b.  in  the  final  report  to  the  F-14D  Program  Manager  as  well  as  the 
Commander,  Operational  Test  and  Evaluation  Force.  Accordingly,  we  ask  that 
management  reconsider  its  position  in  response  to  the  final  report. 

Director,  Operational  Test  and  Evaluation,  Conunents.  The  Director  stated 
that  resolution  of  previous  effectiveness  and  suitability  issues  is  not  a 
pre-condition  of  fielding  the  jammer  in  F-14D  aircraft.  He  suggested  deleting 
ASPJ  measures  of  effectiveness  in  draft  report  Recommendations  l.b.  (ASPJ 
mission  effectiveness  improvement)  and  l.c.  (ASPJ  BIT  performance 
improvement)  and  replacing  them  with  Recommendation  l.b.,  a  new  ASPJ 
measure  of  effectiveness  for  suitability. 

Audit  Response.  Based  on  management  comments,  we  deleted  draft  report 
Recommendations  l.b.  and  l.c.  and  rq)laced  them  with  a  new  Recommendation 
l.b.,  addressing  a  new  measure  of  effectiveness  for  suitability,  including  BIT 
performance. 
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2.  We  reconunend  that  the  Director,  Operational  Test  and  Evaluation, 
approve  the  F-14D  Aircraft  Test  and  Evaluation  Master  Plan  only  after  flie 
F-i4D  Program  Manager  and  the  Commander,  Operational  Test  and 
Evaluation,  include  speciflc  Airborne  Self-Protection  Jammer  measures  of 
effectiveness  as  identiffed  in  Recommendation  1. 

Director,  Operational  Test  and  Evaluation,  Comments.  The  Director 
concurred  with  the  intent  of  Recommendation  1,  but  stated  that  the 
recommended  measures  of  effectiveness  were  not  applicable  to  the  scope  and 
planned  outcome  of  the  F-14D/ASPJ  operational  test  phase.  However,  he  stated 
that  it  is  essential  that  the  planned  F-14D  operational  test  characterize  the 
jammer's  discrete  capabilities  and  limitations  when  operating  in  the  F-14D 
aircraft  and  its  mission  environment.  In  this  regard,  he  stat^  that  he  would 
only  approve  the  F-14D  TEMP  and  Test  Plan  if  the  Navy  provides  for 
collection  of: 

o  discrete  effectiveness  data  sufficient  to  determine  that  the  F-14D 
aircraft  is  more  survivable  with  the  jammer  than  without  it  and 

o  discrete  suitability  testing  to  characterize  ASPJ  reliability, 
maintainability,  and  BIT-installed  performance  as  a  component  of  the  F-14D’s 
integrated  EW  and  avionics  suite. 

Audit  Response.  Management's  comments  are  partially  responsive  to  the  intent 
of  the  recommendation.  We  applaud  the  Director's  decision  not  to  approve  the 
F-14D  TEMP  and  Test  Plan  until  the  Navy  plans  to  collect  discrete  ASPJ 
effectiveness  and  suitability  data.  We  still  believe,  however,  tiiat  the  DoD  and 
Navy  need  to  establish  objective  and  meaningful  ASPJ  measures  of  effectiveness 
in  the  F-14D  TEMP  and  Test  Plan.  We  do  not  advocate  that  the  recommended 
ASPJ  measures  of  effectiveness  correlate  directly  to  the  ASPJ  measures  of 
effectiveness  established  for  earlier  ASPJ  operational  tests  but  that  they  are 
adequate  to  determine  whether  or  not  the  jammer  makes  a  positive  contribution 
to  F-14D  aircraft  survivability  and  whether  the  demonstrated  ASPJ  BIT 
performance  is  satisfactory.  Accordingly,  we  ask  that  the  Director  reconsider 
his  position  in  response  to  the  final  rqx)rt. 

Note:  We  also  made  appropriate  changes  to  the  report  based  on  specific 
management  comments  made  on  statements  in  the  draft  report.  The  complete 
text  of  management  comments  is  in  Part  IV. 
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Part  III  -  Additional  Information 


Appendix  A.  Letter  From  Senator  Pryor 
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Bnitd  States  Senate 

WASHINGTON.  DC  20510-0402 


i^ril  14,  1994 


AGMCULTUOE.  NUTMTtON.  AND 
OOKSTOV 
FWMMCe 

GOVENNMENTAl  AFFAIRS 
SFiClAL  COMMirrCE  ON  AQINC 


Mr.  Derek  J.  Vender  Schaaf 
Deputy/Acting  Inspector  General 
U.S.  Department  of  Defense 
400  Army -Navy  Drive 
Room  1000 

Arlington,  VA  22202 
Dear  Mr.  Vender  Schaaf: 

I  am  writing  to  ask  that  your  office  conduct  a  timely  audit  of 
the  Department  of  Defense's  current  plans  relative  to  the  Airborne 
Self -Protection  Jammer  (ASPJ)  program. 

As  you  know,  the  ASPJ  program  has  a  long,  troubled  history. 
This  poorly  managed  program  was  constantly  plagued  by  cost  overruns 
and  inadequate  performance.  In  1992,  ASPJ  was  cancelled  after  it 
failed  operational  test  and  evaluation  on  the  Navy's  F/A-18 
aircraft.  Unfortunately,  during  the  16  year  life  of  ASPJ,  the  Navy 
contradicted  our  "fly  before  you  buy"  pledge  by  purchasing  95  ASPJ 
jammers  before  testing  had  justified  such  purchases. 

I  am  disturbed  by  the  DoD's  plan  to  test  and  install  the  95 
existing  ASPJ' s  on  our  F-14D  aircraft.  As  you  may  know,  the  Navy  is 
currently  updating  its  F-14D  Test  and  Evaluation  Master  Plan  (TEMP) 
to  include  a  limited  ASPJ  evaluation.  I  was  recently  informed  that 
these  operational  tests  will  not  proceed  until  a  Director, 
Operational  Test  and  Evaluation  (DOT&E)  has  been- confirmed  by  the 
U.S.  Senate  and  allowed  proper  input  in  this  process. 

Given  the  continuing  absence  of  a  test  director,  I  find  it 
necessary  to  ask  you  to  audit  the  U.S.  Navy  and  its  plans  to  test 
and  install  these  controversial  radar  jammers  in  our  F-14D  aircraft. 
Specifically,  I  would  like  you  to  assess  the  adequacy  of  the  planned 
F-14D  tests  in  determining  whether  ASPJ  would  protect  our  Navy 
pilots,  and  also,  whether  these  tests  would  address  ASPJ's 
inadequacies  that  were  exposed  during  previous  operational  tests.  I 
also  hope  your  audit  would  address  the  impact  of  planned  ASPJ  P-14D 
testing  on  the  Navy's  plans  for  future  upgrades  to  that  aircraft. 

Furthermore,  I  am  concemedi  about  the  DoD  assisting  in  the 
foreign  sales  of  the  ASPJ  to  our  allies.  As  you  may  know,  current 
Defense  Department  policy  prohibits  the  production  of  additional 
ASPJ  radar  jammers.  I  would  like  your  investigation  to  include  an 
analysis  of  whether  the  DoD's  support  for  the  integration  of  ASPJ 
through  the  Foreign  Military  Sales  program  contradicts  this  policy 
of  no  new  ASPJ  production. 
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Finally,  when  the  Air  Force  abandoned  ASPJ  in  1989,  the  DoD 
allowed  the  Services  to  act  independently  in  their  efforts  to  meet 
aircraft  self -protect ion  requirements.  I  have  grown  increasingly 
concerned  about  the  lack  of  corporate  thinking  in  this  area . 

Since  the  ASPJ  program  was  cancelled  in  1992,  the  DoD  has  yet 
to  reevaluate  its  self -protection  needs  for  all  Air  Force  and  Navy 
aircraft  in  an  effort  to  reassess  how  the  military  Services  might 
work  cooperatively  to  save  time  and  money. 

1  would  like  your  opinion  on  the  adequacy  of  the  DoD^  s  efforts 
to  find  cost-effective  systems  for  maximum  common  use  on  existing 
and  next  generation  Air  Force  and  Navy  aircraft .  Please  provide 
suggestions  on  how  the  DoD  might  best  achieve  this  goal . 

I  sincerely  hope  your  audit  will  be  completed  expeditiously. 
Please  be  prepared  to  update  my  office  of  your  progress  by  June  1, 
1994.  If  you  have  any  questions  regarding  this  request,  please 
contact  my  office  at  (202)  224-2353. 

Sincerel^fcv 


Sincerel^n^ 


David  Pryor 


Appendix  B.  DoD  Efforts  to  Identify  Cost- 

Effective  Self-Protection  Systems 
for  Common  Use 


We  reviewed  on-going  Navy  and  Air  Force  aircraft  development  efforts  to 
determine  DoD  efforts  to  develop  cost-effective  self-protection  systems  for 
common  use  on  Navy  and  Air  Force  aircraft.  These  efforts  are  primarily 
focused  on  the  next  generation  of  Navy  and  Air  Force  aircraft  needed  as 
replacements  for  the  AV-8B,  F/A-18,  F-14,  and  F-16  aircraft,  and,  later,  the 
F-  15E  and  F- 1 1 1  aircraft. 


Aircraft  Programs 


F-22  Aircraft  Program.  The  F-22  aircraft  program  established  the  Joint 
Integrated  Avionics  Working  Group  (Joint  Worlang  Group)  to  develop  common 
avionics  across  new  aircraft  developments.  As  a  result,  the  Integrated 
Electronic  Warfare  System  will  be  developed  for  use  in  the  F-22  aircraft.  It 
will  provide  an  advanced  EW  capability  to  include  self-protection  jamming  for 
the  F-22  and  future  aircraft  by  eliminating  redundant  hardware  and  using 
common  electronic  modules.  The  Joint  Advanced  Strike  Technology  (JAST) 
program  is  structured  to  capitalize  on  the  F-22  program  effort  to  define  a 
common  avionic  architecture  for  its  flying  demonstrators. 

Advanced  Short-Takeoff  and  Vertical  Landing  Program  (ASTOVL).  The 
ASTOVL  program  is  a  joint  effort  between  the  Navy,  the  Advanced  Research 
Project  Agency,  and  the  National  Aeronautics  and  Space  Administration  to 
demonstrate  technologies  required  to  construct  and  fly  an  ASTOVL 
demonstrator  aircraft  by  FY  1999.  The  ASTOVL  Program  will  investigate  the 
feasibility  of  designing  a  single  lightweight,  affordable  aircraft  to  conduct 
missions  currently  performed  by  the  AV-8B,  F-16,  and  F/A-18  aircraft.  In 
FY  1995,  the  JAST  Program  Office  will  ev^uate  the  ASTOVL  aircraft  as  a 
candidate  for  one  of  the  flying  concept  demonstrators  to  satisfy  joint  Military 
Department  requirements.  As  a  JAST  flying  demonstrator,  the  aircraft  would 
be  subject  to  the  same  EW  system  commonality  requirements  as  other  emerging 
prototype  aircraft. 

Joint  Advanced  Strike  Technology  Program.  The  JAST  program  is  designed 
to  pursue  technology  that  future  U.S.  combat  aircraft  can  use.  In  terms  of 
avionics  and  software,  such  as  aircraft  self-protection  systems,  the  JAST 
program  plans  to  identify  critical  technologies  in  the  area  of  avionics 
architecture  definition  that  will  capitalize  on  efforts  of  the  F-22  Joint  Working 
Group.  According  to  the  Bottom-Up  Review,  the  JAST  program  will  focus  on 
developing  common  components  for  future  engines,  avionics,  ground  support. 
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Common  Use 


training,  munitions,  and  advanced  mission  planning.  DoD  expects  to  achieve 
commonality  by  using  joint  requirements,  proven  technology,  and  a  requirement 
to  make  the  resulting  aircraft  affordable. 


Conclusion 


The  DoD  has  on-ping  Navy  and  Air  Force  aircraft  development  efforts  to 
develop  cost-effective  self-protection  systems,  among  other  aircraft  components, 
for  common  use  on  Navy  and  Air  Force  aircraft.  The  JAST  program  will 
produce  two  demonstrator  aircraft  with  different  technologies  that  could  be 
incorporated  in  next-generation  Navy  and  Air  Force  aircraft.  Also,  the  F-22 
aircraft  and  ASTOVL  programs  will  make  significant  technology  contributions 
to  the  two  demonstrator  aircraft  that  will  evolve  from  the  JAST  program.  From 
the  aircraft  technology  demonstrations,  DoD  expects  to  achieve  commonality  in 
next-generation  combat  aircraft  systems,  including  self-protection  systems. 
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Appendix  C.  Prior  Audits  and  Other  Reviews 


(^neral  Accounting  Office  (GAO)  Report  No.  GAO/NSIAD  94-202  (OfHce 
^  the  Secretary  of  Defense  [OSD]  Case  No.  9704),  "Use  of  FMS  in 
mposed  Commercial  Sale  of  Airborne  Self-Protection  Jammer,"  June  16, 
1994.  ^e  GAO  reported  that  DoD's  decision  to  permit  the  foreign  military 
sale  of  ASPJ  software  was  not  consistent  with  DoD  statements  in  congressional 
Mrrespondence  that  the  FMS  sale  of  the  ASPJ  would  not  be  allowed. 
However,  the  GAO  stated  that  DoD’s  decision  to  permit  the  sale  of  the  ASPJ's 
Mftware  through  the  FMS  program  did  comply  with  the  USD(A&T)’s  policy  on 
FMS  because  the  sale  was  approved  based  on  national  security  and  technical 
concerns.  Further,  GAO  stated  that  DoD's  decision  did  not  raise  any  U.S. 
financial  liability  or  national  security  concerns.  Specifically,  the  ASPJ  software 
^e  would  not  make  the  U.S.  Government  responsible  for  the  performance  of 
the  commercially  procured  ASPJ  hardware.  In  addition,  the  GAO  stated  that 
the  U.S.  Government  would  hEve  no  cost  implications  should  disputes  arise 
b^use  FMS  buyers  are  responsible  for  all  program  costs.  GAO  agreed  with 
DoD  that  the  sale  of  software  through  FMS  procedures  is  a  standard  practice. 
The  report  did  not  contain  recommendations. 


GAO  Report  No.  GAO/NSIAD  92-103  (OSD  Case  No.  8943),  "Electronic 
Warfare:  Established  Criteria  Not  Met  for  Airborne  Self-Protection 
Jammer  Production,"  March  23,  1992.  The  GAO  reported  that  the  ASPJ  did 
not  the  Defense  Acquisition  Board  (DAB)  established  reliability  criterion 
for  further  production.  Before  the  production  decision,  the  GAO  stated  that  the 
Navy  changed  the  DAB-approved  reliability  criterion  to  exclude  system  failures 
EttnbutH.blc  to  software  errors  after  system  failures  occurred  during  the 
reliability  growth  testing.  The  GAO  stated  that  the  DAB  knew  at  the  June  24 
;  otJt’  that  the  Navy  had  changed  the  criterion  for  measuring 

ASPJ  rehability  but  had  allowed  the  ASPJ  program  to  continue  production.  If 
software-induced  failures  had  been  in  the  reliability  test  results,  the  ASPJ  would 
have  failed  the  test  by  a  large  margin.  Further,  the  GAO  reported  that  ASPJ 
reliability  growth  toting  conducted  after  the  DAB  showed  that  ASPJ's  software 
problems  were  continuing.  The  report  did  not  contain  recommendations. 

A^istant  Inspector  General  for  Auditmg,  DoD,  Report  No. 
W-^9,  'The  Airborne  Self-Protection  Jammer  Program  as  Part  of  the 
Defense  Acquisition  Board  Review  Process  -  FY  1993,"  March  22  1993 
This  classified  report  concluded  that  the  ASPJ  did  not  meet  user  ne^s  and 
sta^  that  a  cost  and  operational  effectiveness  analysis  (COEA)  had  not  been 
^rformed  to  support  the  planned  Milestone  HI,  Production  and  Development 
decision.  ^  ’ 

The  report  recommended  to  the  USD(A&T)  that 

o  the  ASPJ  program  be  canceled, 

o  the  Navy  and  Air  Force  prepare  a  COEA, 
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o  a  DAB  program  review  be  convened  to  determine  the  best  course  of 
action  for  fulfilling  overall  DoD  operational  requirements  for  aircraft  self¬ 
protection,  and 

o  the  Navy  end  procurement  of  ASPJ  systems  as  low-rate  initial 
production  and  terminate  production. 

In  response  to  the  report,  the  USD(A&T)  directed  the  Navy  to  cancel  ASPJ 
production  contracts  that  implemented  Recommendations  1.  and  4.  Concerning 
Recommendations  2.  and  3.,  the  USD(A&T)  agreed  to  review  dl  possible 
alternatives  to  fulfill  the  urgent  requirement  for  self-protection  on  the  F/A-18 
aircraft,  including  new  systems,  and  that  any  decision  concerning  alternatives 
would  be  based  on  a  new  COEA.  The  COEA  will  be  conducted  in  FY  1995. 
Management  comments  also  noted  that  the  Navy  and  the  Air  Force  are  already 
joint  participants  in  a  classified  program  on  aircraft  self-protection. 
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Appendix  D 


Recommendation 

Reference 


1. 


2. 


Summary  of  Potential 
Resulting  From  Audit 


Description  of  Benefit 


Program  Results.  Will  ensure  that 
the  Navy  establishes  measures  of 
effectiveness  needed  for  the  testers 
to  determine  whether  the  ASPJ 
contributes  to  F-14D  aircraft 
survivability. 

Program  Results.  Will  ensure  that 
the  Navy  begins  the  F-14D 
aircraft-dedicated  operational  test 
and  evaluation  only  after 
ASPJ-specific  measures  of 
effectiveness  are  established. 


Benefits 


Amount  and/or 
Type  of  Benefit 


Nonmonetary. 


Nonmonetary. 
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Appendix  £.  Organizations  Visited  or  Contacted 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology,  Washington,  DC 
Director,  Acquisition  Program  Integration,  Washington,  DC 
Director,  Test  and  Evaluation,  Washington,  DC 

Depu^  Director,  Electronic  Warfare,  Washington,  DC 
Principal  Deputy  Assistant  Secret^  of  Defense  (Dual  Use  Technology  and 
International  Programs),  Washington,  DC 
Director,  Operational  Test  and  Evaluation,  Washington,  DC 
Deputy  Director,  Weapon  Systems  Division,  Defense  Security  Assistance  Agency, 
Arlington,  VA 


Office  of  the  Secretary  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition), 
Washington,  DC 

Joint  Advanced  Strike  Technology  Program  Office,  Arlington,  VA 
Program  Executive  Officer,  Tactical  Aircraft  Programs,  Arlington,  VA 
F-14D  Program  Office,  Arlington,  VA 
F/A-18  Program  Office,  Arlington,  VA 
Tactical  Air  Electronic  Warfare  Program  Office,  Arlington,  VA 
Naval  Air  Systems  Command,  Arlington,  VA 
Navy  International  Program  Office,  Arlington,  VA 
Commander,  Operational  Test  and  Evaluation  Force,  Norfolk,  VA 
Air  Test  and  Evaluation  Squadron  Four,  Point  Mugu,  CA 
Weapons  Division,  Naval  Air  Warfare  Center,  Point  Mugu,  CA 


Office  of  the  Secretary  of  the  Air  Force 

Global  Integration  Division,  Deputy  Assistant  Secretary  of  the  Air  Force  for 
International  Affairs,  Arlington,  VA 


Other  Government  Organization 

General  Accounting  Office,  Washington,  DC 
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Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Director,  Test,  Systems  Engineering  and  Evaluation 
Director,  Tactical  Warfare  Programs 
Deputy  Under  Secretary  of  Defense  (Acquisition  Reform) 
Under  Secretary  of  Defense  (Comptroller) 

Director,  Defense  Security  Assistance  Agency 
Director,  Operational  Test  and  Evaluation 
Assistant  to  the  Secretary  of  Defense  (Public  Affairs) 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management) 

Assistant  Secretary  of  the  Navy  (Research,  Development,  and  Acquisition) 
Joint  Advanced  Strike  Technology  Program  Office 
Program  Executive  Officer,  Tactical  Aircraft  Programs 
F-14D  Program  Office 

Tactical  Air  Electronic  Warfare  Program  Office 
Naval  Air  Systems  Command 
Navy  International  Program  Office 
Commander,  Operational  Test  and  Evaluation  Force 
Auditor  General,  Department  of  the  Navy 


Other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 

Director,  Defense  Logistics  Agency 

Director,  National  Security  Agency 

Inspector  General,  Central  Imagety  Office 

Inspector  General,  Defense  Intelligence  Agency 

Inspector  General,  National  Security  Agency 

Director,  Defense  Logistics  Studies  Information  Exchange 


Non-Defense  Federal  Organizations 

Office  of  Management  and  Budget 

U.S.  General  Accounting  Office,  National  Security  and  International  Affairs  Division, 
Technical  Information  Center 
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Chairman  and  Ranking  Minority  Member  of  each  of  the  following  Congressional 
Committees  and  Subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 

Senate  Committee  on  Armed  Services 

Senate  Committee  on  Governmental  Affairs 

House  Committee  on  Appropriations 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 
House  Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal 
Justice,  Committee  on  Government  Reform  and  Oversight 
House  Committee  on  National  Security 
Senator  David  Pryor 
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Part  rV  -  Management  Comments 


Director,  Operational  Test  and  Evaluation, 

Conunents 


OFFICE  OF  THE  SECRETARY  OF  DEFENSE 
WASHINGTON.  DC  2030)1700 


?  0  KC  HM 


MEMORANDUM  FOR  THE  INSPECTOR  GENERAL,  DOD 


SUBJECT:  Report  on  the  Congressionally  Requested  Review  of  the  Airborne  Self 
Protection  Jammer  (Project  No.  4AE-5036) 


I  have  reviewed  the  findings  and  recommendations  in  the  October  31. 1994  draft  report 
and  offer  the  following  comments: 

1.  I  will  only  apiHX>ve  the  appropriate  F*14D/ASPJ  TEMP  and  test  plan  if  they  provide 
for  collection  of  discrete  effectiveness  data  sufficient  to  determine  the  F-14D  is  more 
survivable  with  the  jammer  than  without  it.  This  is  fully  compliant  with  the  Department's 
guidance  to  the  Navy  regarding  preconditions  for  a  favorable  ASPJ  fielding  decision. 
Furthermore,  I  will  ensure  the  F-14D/ASPJ  OT&E  plan  outlines  discrete  suitability  testing 
sufficient  to  characterize  ASPJ  reliability,  maintainability,  and  built>in>test  installed 
performance  as  a  component  of  the  F'14D's  integrated  electronic  warfare  and  avionics 
suite. 

2.  While  I  concur  with  the  overall  intent  of  the  Recommendations  For  Corrective 
Action  (page  12,  para  1),  I  am  concerned  that  your  proposed  Airborne  Self-Protection 
Jammer  measures  of  effectiveness  ate  not  applicable  to  the  scope  aiul  planned  outcome  of 
the  F-14D/ASPJ  OT&E  phase.  I¥evious  ASPJ  operational  testing  has  been  adequate  to 
determine  that  the  jammer  is  neither  effective  nor  suitable  when  evaluated  against  user 
derived  criteria.  The  Navy  admowledges  the  systems  already  delivered  have  performance 
limitations,  but  the  Service  believes  ASPJ  can  still  contribute  an  important  incremental 
improvement  to  F-14D  combat  survivability  compared  to  the  no  jammer  alternative.  Since 
the  Navy  plans  to  install  existing  systems,  it  was  never  envisioned  that  the  F-14D/ASPJ 
operational  evaluation  would  favorably  resolve  known  performance  deficiencies.  It  is 
essential,  however,  that  planned  OT&E  characterize  die  jammer's  discrete  capabilities  and 
limitations  when  operating  in  the  F-14D  and  its  mission  environment 

3.  Although  the  Navy  has  no  plans  to  procure  additional  ASPJ  systems,  a  robust, 
disciplined  F-14D/ASPJ  operational  test  is  needed  to  support  an  informed  fleet  fielding 
decision.  TheF-14D0T^  will  not  be  a  re-test  of  ASPJ.  (Quantifiable  test  results  are 
applicable  only  to  the  F-14D  installed  configuration  and  will  not  supplant  determirutions 
reached  during  the  previous  F/A-18/ASPJ  OT&E  phases.  Since  the  F-14D's  primary 
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mission  and  combat  operating  environments  are  different  than  the  F/A'18*s  intended 
concept  of  operadensrtest  mission  profiles  and  specific  threat  laydowns  roust  be  tailored 
to  the  F-14D's  primary  mission.  As  a  result,  some  test  methodology  and  evaluation 
criteria  used  in  previous  ASPJ  OT&E  will  not  ^)ply  to  the  F-14D  test 

4.  While  F-14D/ASPJ  OT&E  must  be  adequate  to  detennine  whether  or  not  the 
jammer  makes  a  positive  contribution  to  F-14D  combat  survivability,  results  should  not  be 
used  to  establish  the  efficacy  of  ASPJ  when  compared  to  other  aircraft  types  or  models. 
F'14D  installed  system  bmlMn-test  and  single  engagement  mission  effectiveness  results 
can  not  be  directly  compared  to  outcomes  of  prior  F/A-IS/ASPJ  testing  due  to  different 
hardware  installation,  antenna  coverage,  aircnfft  radar  cross  section,  software  integration, 
etc.  In  our  view,  the  F-14D  results  should  only  be  compared  with  similar  trends  observed 
during  the  F/A- 18  testing  to  increase  confidence  in  the  observed  performance. 


Recommended  Changes; 

1.  (re:  pages  10-11,  para  4,  Comparison  of  ASPJ  Test  Results)  Delete  all  reference  to 
comparison  test  outcomes.  As  noted  above,  F*14D  and  F/A-18  have  unique  ASPJ 
hardware  and  software  integration  schemes  which  make  it  extremely  difficult  to  isolate  the 
actual  cause  of  observed  performance  variance.  Furthermore,  differences  in  expected 
concept  of  operations  reduce  the  operational  relevance  of  any  direct  performance 
comparisons. 

2.  (re:  page  12,  para  1,  Recommendation  for  Corrective  Action)  Delete  the  current 
sub  paragraphs  b&c.  Insert  a  new  sub  paragraph  b  to  read: 

b.  Airborne  Self^Protection  Jammer  suitability. 


1  appreciate  this  opportunity  to  respond  to  your  preliminary  findings  and 
recommendations,  my  staff  P()C  for  the  Airborne  &lf-Prolection  Jammer  program  is 
Colonel  Gerry  Christeson,  (703)697>3891. 


Philip  E.  Coyle 
Director 


Naval  Commander,  Operational  Test  and 
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DEPARTMENT  OP  THE  NAVY 

COMMANDED  OfCTATIONAL  TEST  AND  EVALUATION  FOftCE 
7970  DIVEN  STREET 
NORFOLK.  VMOINU  23B06-1498 

04  Jan  95 

MEMORANDUM 

From:  Commander,  Operational  Test  and  Evaluation  Force 
To:  Office  of  the  Inspector  General,  Department  of  Defense 

Subj:  DOD/IG  DRAFT  REPORT  ON  THE  CONGRESS lONALLY  REQUESTED 

REVIEW  OF  THE  AIRBORNE  SELF-PROTECTION  JAMMER  (PROJECT  NO. 
4AE-5036) 

Ref:  (a)  DoD/IG  report  of  the  F-14D/ASPJ  Test  Plan  of  31  Oct  94 

(b)  CNO  Memorandum  Ser  N880C1/4U65241  of  18  Nov  94 

Enel:  (1)  Detailed  Comments  to  the  DoD/IG  report  of  the 

F-14D/ASPJ  Test  Plan  of  31  Oct  94 

1.  As  requested  in  reference  (a),  enclosure  (1)  is  submitted  for 
your  review. 

2.  COMOPTEVFOR  has,  for  some  time,  advocated  a  more  logical  and 
fiscally  responsible  approach  in  testing  the  F-14D  and  associated 
subsystems.  To  this  end,  the  following  will  clarify  current 
efforts.  The  F-14D,  along  with  the  IRST  and  ASPJ  subsystems, 
have  been  purchased  and  no  further  Milestone  III  acquisition 
decisions  remain.  Aircraft  and  aircrews  are  now  actively 
deployed  on  the  second  extended  deployment,  participating  in  real 
world  contingency  operations.  However,  due  to  ASPJ  issues 
regarding  testing  and  integration,  F-14D  software  and  tactical 
development  continues  to  incur  significant  delays. 

3.  As  a  result,  CC»!0PTEVF0R  has  orchestrated  F-14D  test 
initiatives  in  support  of  continuing  fleet  deployments.  In 
September  1994,  a  Quick  Reaction  Assessment  (QRA)  phase  which 
focused  primarily  on  the  ECM  capabilities  of  OFP  DOl  was 
conducted.  As  a  result  of  this  effort,  as  well  as  fleet  and 
other  exercise  data  gathered  to  date,  OFP  DOl  can  now  be  closed 
out  and  follow-on  software  development  continued.  Reference  (b) 
endorses  this  approach  and  provides  further  guidance  in  the 
direction  of  the  F-14D  program.  ASPJ  testing  is  currently  being 
re-aligned  to  occur  in  conjunction  with  OFP  D02  development/ 
testing  (OT-IIIA) .  COMOPTEVFOR  is  working  closely  with  PMA-241 
and  the  F-14D  N-88  sponsor  to  update  both  the  TEMP  (Rev.  A)  and 
Test  Plan  to  include  specific  ASPJ  COI's.  COMOPTEVFOR  believes 
that  this  effort  is  in  the  best  interest  of  the  fleet  and  will 
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Subj:  DOD/IG  DRAFT  REPORT  ON  THE  CONGRESS lONALLY  REQUESTED 

REVIEW  OF  THE  AIRBORNE  SELF- PROTECT  I  ON  JAMMER  (PROJECT  NO. 
4AE-5036) 

allow  the  Navy  to  fully  evaluate  ASPJ  enhancements  to  the  F-14D 
and  support  an  informed  decision  regarding  future  ASPJ 
deployment . 


for  Air  Warfare 


Chief  of  Staff 
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DETAILED  CCMMENTS  OF  THE  DOD/IG  REVIEW 

1.  Page  li.  Summary  of  Recommendations. 

Issue  (1) :  The  first  paragraph  recommends  that  "the  Commander, 
Operational  Test  and  Evaluation  Force,  revise  the  F-14D  Aircraft 
Test  and  evaluation  Test  Plan  to  include  ASPJ  measures  of 
effectiveness, ..." 

Comment  (1) :  As  a  result  of  several  Test  Planning  Working 
Groups,  the  TEMP  was  updated  to  include  three  measures  of 
effectiveness  (Missile  Lethality,  Shot  Degradation  and  Shot 
Denial) .  COMOPTEVFOR  believes  that  the  inclusion  of  these 
additional  measurements  will  allow  sufficient  ASPJ  performance 
data  to  be  collected  during  the  testing  period.  It  is  the  intent 
of  CCMOPTEVFOR  to  then  make  specific  comments  and  recommendations 
concerning  ASPJ  performance  in  the  OT-IIIA  final  report. 

Comment  (2)  :  For  clarity  purposes,  it  should  be  noted  that  while 
CCWOPTEVFOR  does  provide  the  Part  IV  and  V  inputs  to  the  TEMP, 
these  inputs  are  based  upon  the  requirements  established  by  the 
PMA  in  Part  I  of  the  document.  Any  deviation  from  the 
requirements  established  in  Part  I  of  the  TEMP  must  be  approved 
by  the  PMA.  While  CCMOPTEVFOR  can  strongly  advocate  desired 
changes,  it  is  ultimately  the  responsibility  of  the  PMA  to  direct 
revisions. 

Recommendation:  Replace  the  paragraph  with  the  following: 

"We  recommend  that  PMA-241  revise  the  Test  and  Evaluation 
Master  Plan  and  F-14D  Aircraft-Dedicated  Operational  Test  and 
Evaluation  Test  Plan  to  include  specific  Airborne  Self- Protection 
Jammer  measures  of  effectiveness." 

Issue  (2);  The  second  paragraph  recommends  that  "the  Director, 
Operational  Test  and  Evaluation,  withhold  approval  of  the  F-14D 
Aircraft  Test  and  Evaluation  Master  Plan  until  the  Commander, 
Operational  Test  and  Evaluation  Force,  makes  recommended 
revisions. " 

Comment:  As  noted  above,  for  clarity  purposes  it  should  be 
pointed  out  that  the  program  office  {PMA-241)  is  responsible  for 
revision  to  the  TEMP. 

Recommendation:  Delete  "...until  the  Commander,  ^erational  Test 
and  Evaluation  Force,  makes  the  recommended  revisions."  Replace 
with  "..until  the  recommended  revisions  are  made  by  the  F-14 
program  office  (PMA-241)." 

2.  Page  8.  Fourth  paragraph.  ASPJ  Improvements. 

Issue:  Statement  regarding  recent  improvements  to  the  ASPJ  that 
says  "incorporating  an  improved  ALR-67  radar  warning  receiver  to 
the  ASPJ/" 

Comment:  ASPJ  performs  threat  ID  and  direction  finding 
internally  only.  ALR-67  improvements  are  independent  of  ASPJ 
upgrade  efforts  -  there  is  no  multi-sensor  correlation  capability 
(MSI)  between  the  two  subsystems. 

End  (1) 
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Reconanendation:  This  statement  should  be  deleted. 

3.  Page  10.  Survivability  -  Second  paragraph  "Measures  of 
Effectiveness"  states  that  the  F-14D  Test  Plan  did  not  include  an 
ASPJ  specific  survivability  measure  of  effectiveness  similar  to 
the  one  established  for  the  1992  ASPJ  OPEVAL. 

Comments:  The  F-14D  Test  Plan  contains  three  measures  of 
effectiveness  that  will  allow  CC»dOPTEVFOR  to  specifically  address 
ASPJ  effectiveness.  The  MOE*s  are: 

-  shot  denial 

-  missile  lethality 

-  shot  degradation 

These  MOE's  are  similar  to  the  same  component  probabilities  used 
in  the  ASPJ  OPEVAL  MOEs.  The  data  collected  will  be  similar  to 
that  collected  in  the  ASPJ  OPEVAL.  In  addition,  COMOPTEVFOR 
believes  that  sufficient  performance  data  will  be  collected  to 
support  specific  F-14D/ASPJ  comments  and  recommendations  in  the 
OT-IIIA  final  report. 

Recommendation;  Clarification  of  COMOPTEVFOR  testing  methodology 

4.  Page  11.  Measuring  Improvements  in  ASPJ  Built-in-test 

Capability. 

Issue:  DoD/IG  report  states  in  paragraph  2  that  "an  ASPJ  measure 
of  effectiveness  for  BIT  performance  is  necessary  so  the  testers 
can  objectively  comment  on  ASPJ  Bit  performance..." 

Comment:  The  Test  Plan  includes  measures  of  suitability  for 
system  and  subsystem  BIT  performance,  including  ASPJ.  Although 
no  specific  subsystems  are  mentioned,  BIT  performance  will  be 
measured  for  all  applicable  avionic  subsystems.  As  a  subsystem 
of  the  aircraft,  ASPJ  is  included  in  the  list  of  avionics  work 
unit  codes  (WUC)  and  therefore  will  be  evaluated  for  operational 
suitability. 

Recommendation:  Clarification  of  CC»!OPTEVFOR  testing  methodology 

5.  Page  11.  Conclusion. 

Issue:  The  second  sentence  states  that  objective  ASPJ  MOEs 
should  be  included  for  determining  "ASPJ  mission  effectiveness". 
This  term  was  not  previously  defined  and  no  finding  was  mentioned 
in  the  DoD/IG  report.  It  should  be  pointed  out  that  CCH40PTEVF0R 
fully  intended  to  measure  ASPJ  mission  effectiveness  within  the 
currently  existing  "Aircraft  Survivability"  COI.  ASPJ  specific 
MOEs,  noted  in  paragraph  3  of  this  enclosure,  may  better  allow 
COMOPTEVFOR  to  make  more  specific  determinations  as  to  ASPJ's 
positive  contribution  to  F-14D  aircraft  survivability,  ASPJ 
mission  effectiveness,  and  ASPJ  BIT  performance. 

Recommendations : 

(1)  Clarification  of  CCWOPTEVFOR  testing  methodology. 

(2)  This  statement  should  be  deleted  since  ASPJ  mission 
effectiveness  will  be  reported  under  aircraft  survivability. 
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6,  Page  12.  Recommendations  for  Corrective  Action. 
Recommendation  1. 

Issue  (1):  The  paragraph  recommends  that  CC»dOPTEVFOR  revise  the 
TEMP. 

Comment  (1):  The  responsibility  for  updating  the  TEMP  resides 

with  the  program  office  (PMA-241) . 

Recommendation :  Clarity 

Issue  (2) ;  Recommendation  to  include  specific  ASPJ  MOEs 
Comment  (1) :  A  Critical  Operational  Issue  (COI)  was  recommended 
by  the  Test  Planning  Working  Group  and  supported  by  reference 
(b) .  The  COI  "Is  the  F-14D  more  survivable  with  ASPJ  as  a  part 
of  the  EW  suite  than  without  it?"  will  be  included  in  both  the 
updated  TEMP  (Rev  A)  and  Test  Plan. 

Recommendation:  Replace  the  first  sentence  with  the  following: 

"  We  recommend  that  the  Navy  revise  the  Test  and  Evaluation 
Master  Plan  and  F-14D  Aircraft-Dedicated  Operational  Test  and 
Evaluation  Test  Plan  to  include  specific  Airborne  Self-Protection 
Jammer  measures  of  effectiveness  for  determining:...," 
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